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Objectives: This study aims to determine the risk factors predicting rehospitalization by comparing three models and selecting the 
most successful model. 

Methods: In order to predict the risk of rehospitalization within 28 days after discharge, 1 1 951 inpatients were recruited into this study 
between January and December 2009. Predictive models were constructed with three methods, logistic regression analysis, a decision 
tree, and a neural network, and the models were compared and evaluated in light of their misclassification rate, root asymptotic stan- 
dard error, lift chart, and receiver operating characteristic curve. 

Results: The decision tree was selected as the final model. The risk of rehospitalization was higher when the length of stay (LOS) was 
less than 2 days, route of admission was through the out-patient department (OPD), medical department was in internal medicine, 
10th revision of the International Classification of Diseases code was neoplasm, LOS was relatively shorter, and the frequency of OPD 
visit was greater. 

Conclusions: When a patient is to be discharged within 2 days, the appropriateness of discharge should be considered, with special 
concern of undiscovered complications and co-morbidities. In particular, if the patient is admitted through the OPD, any suspected 
disease should be appropriately examined and prompt outcomes of tests should be secured. Moreover, for patients of internal medi- 
cine practitioners, co-morbidity and complications caused by chronic illness should be given greater attention. 
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INTRODUCTION 

Readmission refers to being hospitalized again after being 
discharged and is significant for two reasons: quality and cost 
of health care [1]. In other words, readmission reflects relative- 
ly low quality [2] and also has negative social impacts [3-5]. 
Readmission is being considered as an indicator for evaluating 
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the overall health care environment [6,7]. Australian Council 
on Healthcare Standards and National Health Service of the 
UK use readmission within 28 days after discharge as an indi- 
cator of the quality of health care [8,9]. The readmission rate is 
also one of the indicators of the Hospital Services Evaluation 
Program in Korea [10]. In addition, the rate of readmission was 
chosen as one of the grounds for measuring the quality of 
health care in the National Quality Forum of 2008 and the US 
Centers for Medicare and Medicaid, pointed out readmission 
as a reason for the excessive medical expenses [1 1 ]. 

The US Medicare Payment Advisory Commission stated that 
$12 billion per year is spent on preventable readmissions [4], 
and a study estimated the cost of readmission of Medicare pa- 
tients to be $17.4 billion per year [5]. When the medical costs 
of admitted patients are analyzed, high cost patients, who 
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comprise 15% of all admitted patients, account for 55.3% of 
total medical costs, while the readmitted patients accounted 
for the highest portion, 42.0% [1 2]. 

The frequency of readmission varies depending on the num- 
ber of days after discharge. Research on Medicare patients in 
the US shows that the rate of readmission within 7 days was 
6.2%, 1 5 days 1 1 .3%, and 30 days 1 7.6% [1 3]. In another study, 
patients readmitted within 30 days accounted for 1 9.6% of the 
total number of patients, which was about a fifth of the total 
number of patients [5]. 

As described above, readmission not only degrades the qual- 
ity of health care but also increases medical expenses. Conse- 
quently, it is important to identify and predict the causes of re- 
admission in order to prevent it. In this study, models to pre- 
dict the risk of readmission were constructed, compared, and 
evaluated in order to determine an optimum model. In addi- 
tion, the final model was applied to the research data in order 
to identify patients at risk of readmission in order to help pre- 
vent readmission. 

METHODS 

Materials and Subjects 

The database of patients admitted to a teaching hospital in 
Seoul from January 1 to December 31, 2009, was used. The 
subject of analysis was individual patients, and when a patient 
had been admitted several times, all cases were evaluated in- 
dividually and included in the analysis. Excluding the diseases 
that occupied less than 5% of the sample size among the 22 
disease categories of the 10th revision of the International 
Classification of Diseases (ICD-10) code, 11 951 hospitalized 
patients among a total of 16 347 admitted under the follow- 
ing 8 categories of diseases were the final subjects of analysis: 
"neoplasms,""endocrine, nutritional and metabolic,""circulato- 
ry system,""respiratory system,""digestive system,""musculosk- 
eletal system and connective tissues,""genitourinary system," 
and "symptoms, signs and abnormal clinical and laboratory 
findings, not elsewhere classified." 

Variables 
Dependent variable 

Readmission is a dependent variable for which the period 
between discharge and readmission varies according to the 
researcher. However, governmental agencies in Canada, Aus- 
tralia, the UK, and New Zealand, as well as most recent studies 



use 1 month (28-30 days) as the standard [8,9,14-18]. In this 
study, readmission within 28 days after discharge was used as 
the dependent variable. 

Independent variable 

The risk factors of readmission revealed in previous studies 
are composed mainly of "demographic,""treatment and clini- 
cal," and "health care utilization" factors. Demographic factors 
were sex [17], age [17,19-21], income [22], and level of educa- 
tion [16]. As for age, opinions about its significance differ 
among researchers [20,23-25]. Treatment and clinical factors 
identified in previous research include department of treat- 
ment [17,20], number of comorbidities [22,23], surgery [26,27], 
clinical test results [28], change in the amount of drug dose 
within 48 hours of discharge [20], experience of depression [21], 
and state of mental health [28]. Health care utilization factors 
previously identified were the length of hospitalization [20], 
frequency of hospitalization and use of the emergency room 
(ER) within 6 months prior to the index hospitalization [28], fre- 
quency of hospitalization within 1 year from the index hospital- 
ization [1 9], type of insurance [1 8], and type of ward [27]. 

For this study, risk factors used in the demographic category 
were sex, age, and region of residence; for the treatment and 
clinical category, the risk factors were department of treat- 
ment, premium medical treatment (selecting a specific doc- 
tor), number of comorbidities, number of accompanying treat- 
ments, and whether the patient had surgery; the categories of 
main diagnosis of ICD-10 were included as independent vari- 
ables; for the health care utilization category, type of insurance, 
type of patient room, admission route, length of stay (LOS), 
number of out-patient department visits and hospitalization, 
and useofER within one year of the index hospitalization were 
taken into consideration as independent variables. 

Construction and Validation of the Model 

Each of the three models in this study was constructed us- 
ing a different method. The first model was constructed using 
logistic regression, which is widely used when the dependent 
variable is binomial, and the second was built with a decision 
tree, which decides the most significant independent variable 
in each stage of predicting dependent variables. The third mod- 
el was constructed using a neural network, which is useful for 
classifying and predicting from a large database [29,30]. 

The complete analysis data were divided into 70% training 
data and 30% validation data, and in order to decide the mod- 
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el with the highest predictive power, the misclassifkation rate 
and root asymptotic standard error (ASE) were compared for 
each of the model's training data and validation data. A lift 
chart and receiver operating characteristic (ROC) curve were 
also used [29,30]. The model finally selected was then applied 
to the research data to identify patients with high readmission 
risk. 

RESULTS 

Characteristics of Subjects 

The demographic characteristics of subjects were as follows: 
men account for 51.0%, as for age, 22.6% were in their 60s, 
20.4% in their 50s, and 1 8.4% over 70, and more than half of the 
patients (56.3%) were residents of the city of Seoul (Table 1). 

Treatment and clinical characteristics were as follows: 64.1% 
of the patients were treated in the Department of Internal Medi- 
cine and 93.0% chose premium medical treatment. Among the 
principal diagnoses, diseases of the digestive system com- 
prised 20.6% of patients and neoplasms 1 8.2% (Table 2). 

Health care utilization characteristics were as follows: pa- 
tients with National Health Insurance accounted for 93.9%, 
and most of the patients (87.6%) stayed in rooms with 2 or 
more beds. Hospitalization via outpatient visits comprised 



Table 1 . Demographic characteristics of subjects 





All 


Nonrehos- 
pitalized 


Rehospital- 
ized 


p-value 

(X 2 or 
Mest) 


Sex 










Male 


6096(51.0) 


4133(48.6) 


1963(56.9) 


<0.001 


Female 


5855 (49.0) 


4368(51.4) 


1487 (43.1) 




Age 










0-9 


763 (6.4) 


675(7.9) 


88(2.6) 


<0.001 


10-19 


364 (3.0) 


292(3.4) 


72 (2.1) 




20-29 


675(5.6) 


523(6.2) 


152 (4.4) 




30-39 


1032(8.6) 


805(9.5) 


227 (6.6) 




40-49 


1784(14.9) 


1169(13.8) 


615(17.8) 




50-59 


2440 (20.4) 


1596(18.8) 


844(24.5) 




60-69 


2697 (22.6) 


1698(20.0) 


999(29.0) 




70+ 


2195(18.4) 


1742(20.5) 


453(13.1) 




Residential regions 










Seoul 


6723 (56.3) 


4991 (58.7) 


1732(50.2) 


<0.001 


Gyeonggi/lncheon 


2785 (23.3) 


1914(22.5) 


871 (25.2) 




Others 


2443 (20.4) 


1596(18.8) 


847 (24.6) 




Total 


11 951 (100.0) 


8501 (100.0) 


3450(100.0) 





Values are presented as number (%). 



70.0%, and the average LOS was 8.5 + 11 .2 days. The number 
of outpatient visits within one year before the index hospital- 
ization was 14.7 + 20.0 visits, the number of hospitalizations 
within one year prior to the index admission was 2.5 ± 3.4 times, 
and the use of the ER within one year of index hospitalization 
was nil for 89.5% of patients (Table 3). 

Construction and Validation of Models 

From a comparison of the misclassification rate and root ASE 
to evaluate the predictive power of the three models (Table 4), 
the root ASE of the training data results was 0.385 for regres- 
sion, 0.373 for the decision tree, and 0.384 for the neural net- 
work, while that of the validation data was 0.385 for the re- 
gression, 0.369 for the decision tree, and 0.383 for the neural 
network. Thus, the decision tree showed the highest predic- 



Table 2. Clinical characteristics of subjects 





All 


Nonrehos- 
pitalized 


Rehospi- 
talized 


p-value 

(X 2 or 
Mest) 


Medical departments 










Internal 


7661 (64.1) 


5269(62.0) 


2392 (69.3) 


<0.001 


Surgical 


4290 (35.9) 


3232(38.0) 


1058(30.7) 




Premium medical treatment 








Yes 


11 118(93.0) 


7893(92.8) 


3225 (93.5) 


0.24 


No 


833 (7.0) 


608(7.2) 


225(6.5) 




ICD-10 disease categories 








Neoplasms 


2173(18.2) 


986(11.6) 


1187(34.4) 


<0.001 


Endocrine, nutritional 


813(6.8) 


642(7.6) 


171 (5.0) 




and metabolic 










diseases 










Circulatory system 


1248(10.4) 


1035(12.2) 


213(6.2) 




Respiratory system 


1240(10.4) 


1045(12.3) 


195(5.7) 




Digestive system 


2465(20.6) 


1589(18.7) 


876 (25.4) 




Musculoskeletal sys- 


901 (7.5) 


790 (9.3) 


111 (3.2) 




tem and connective 










tissue 










Genitourinary system 


1114(9.3) 


958(11.3) 


156 (4.5) 




Symptoms, clinical 


1997(16.7) 


1456(17.1) 


541 (15.7) 




findings, etc., else- 
where classified 










No. of comorbidities 


1.5 + 1.1 


1.6 + 1.2 


1.4±0.9 


<0.001 


No. of accompanying 


1.413.1 


1.5±3.2 


1.3 + 2.9 


<0.001 


treatments 










Surgery 










Yes 


4451 (37.2) 


3375(39.7) 


1076(31.2) 


<0.001 


No 


7500(62.8) 


5126(60.3) 


2374 (68.8) 




Total 


11 951 (100.0) 8501 (100.0) 3450(100.0) 





Values are presented as number (%) or mean ± SD. 

ICD-10, 10th revision of the International Classification of Diseases. 
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Table 3. Health care utilization characteristics of subjects 



Insurance 

National Health 11220 
Insurance 

Medical Aid 525 

Other 206 

Ward type 

Private room 1144 

Multi-bedroom 10 469 

Intensive care unit 338 

Route of admission 

Out-patient depart- 8368 
ment 

Emergency room 3583 



Length of stay 


8.5 + 11.2 9.0 + 11.3 7.5± 10.9 


<0.001 


No. of out-patient 
department visits 


14.7 + 20.0 11.7 + 17.5 22.0 + 22.5 


<0.001 


No. of admissions 
within previous 12 
months 


2.5 + 3.4 1.8 + 2.2 4.4+4.8 


<0.001 


Utilization of emergency room within previous 1 2 months 




Yes 


1260(10.5) 828(9.7) 432(12.5) 


<0.001 


No 


10 691(89.5) 7673(90.3) 3018(87.5) 




Total 


11 951 (100.0) 8501 (100.0) 3450(100.0) 





Values are presented as number(%) or mean + SD. 



Table 4. Comparison of statistical prediction models by root 
ASE and misclassification rate 





Logistic 


Decision 


Neural 




regression 


tree 


network 


Root ASE Training data 


0.385 


0.373 


0.384 


Validation data 


0.385 


0.369 


0.383 


Misclassification rate Training data 


0.214 


0.180 


0.214 


Validation data 


0.217 


0.177 


0.211 



ASE, asymptotic standard error. 



tive power for the root ASE. The misclassification rate with the 
model generation data was 0.214 for the regression, 0.180 for 
the decision tree, and 0.214 for the neural network while that 
obtained with the validation data was 0.21 7 for the regression 
analysis, 0.177 for the decision tree, and 0.21 1 for the neural 
network. Thus, the decision tree also showed the highest pre- 
dictive power for the misclassification rate. The lift chart and 
ROC curve, which are widely used to evaluate a given model's 
predictive power, were also used, and from the results, both 
the lift chart (Figure 1) and ROC curve (Figure 2) found the de- 



85 r 



10 20 30 40 50 60 70 80 90 100 
Percentile 

Figure 1. Comparison of statistical prediction models by Lift 
chart. 'Decision tree. 




0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 
1 -Specificity 

Figure 2. Comparison of statistical prediction models by re- 
ceiver operating characteristic curve. 'Decision tree. 



cision tree to have stronger predictive power. From the model 
comparison, the decision tree was chosen in order to predict 
patients with readmission risk. 

Predicting Readmission Risk 

The results of the decision tree evaluated the main riskfac- 



,. Nonrehos- Rehospi- 
II i- j . i- j X or 

pitahzed tahzed j 



(93.9) 


7994 (94.0) 


3226 (93.5) 


0.002 


(4.4) 


346 (4.1) 


179(5.2) 




(1.7) 


161 (1.9) 


45(1.3) 




(9.6) 


932(11.0) 


212(6.1) 


<0.001 


(87.6) 


7242 (85.2) 


3227 (93.5) 




(2.8) 


327 (3.8) 


11 (0.3) 




(70.0) 


5565(65.5) 


2803(81.2) 


<0.001 


(30.0) 


2936 (34.5) 


647(18.8) 





75 
70 
65 
60 

¥ 55 

a 

I 50 
45 
40 
35 
30 
25 



Baseline 
Neural network 
Logistic regression 
Decision tree 
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No 


8501 (71.1) 


Yes 


3450 (28.9) 


Total 


11 951 (100.0) 



T 

Length of stay 



Admission route 
0PD I ER 



Neoplasms 



No 


296 (24.7) 


Yes 


904(75.3) 


Total 


1200(100.0) 



No 


228 (35.1) 


Yes 


66 (64.9) 


Total 


294(100.0) 



No 


927 (48.8) 


Yes 


971 (51.2) 


Total 


1898(100.0) 



Medical department 
Surgical Internal 



1 

Length of stay 





<2 | 




>2 








No 


524(35.1) 




No 


7977 (76.3) 


Yes 


970 (64.9) 




Yes 


2480 (23.7) 


Total 


1494(100.0) 




Total 


10 457(100.0) 



ICD-10 



Others 



No 


7050 (82.4) 


Yes 


1509(17.6) 


Total 


8559(100.0) 



T 

No. ofOPD visit 



<5 



>5 



No 


92(60.9) 


Yes 


59 (39.1) 


Total 


151 (100.0) 



No 


204(19.4) 


Yes 


849 (80.6) 


Total 


1049(100.0) 



No 


256 (30.8) 


Yes 


574 (69.2) 


Total 


830(100.0) 



No 


671 (62.8) 


Yes 


397 (37.2) 


Total 


1068(100.0) 





<10 




>10 


I 




I 


No 


4895 (87.5) 




No 


2155 (72.7) 


Yes 


701 (12.5) 




Yes 


808 (27.3) 


Total 


5596(100.0) 




Total 


2963(100.0) 



Figure 3. Prediction models in rehospitalization. Values are presented as number (%). No, nonrehospitalization; Yes, rehospitaliza- 
tion; OPD, out-patient department; ER, emergency room; ICD-10, 10th revision of the International Classification of Diseases. 



tors of readmission as LOS (standard of categorization: 2 days), 
route of admission, category of principal diagnosis (ICE-10), 
department, LOS (standard of categorization: 5 days), and the 
frequency of outpatient visits, in respective order (Figure 3). 

The most important factor in the first stage was the LOS; if 
the LOS was lower than 2 days, the risk of readmission was 
64.9%. The important factor in the next stage was route of ad- 
mission, where hospitalization via outpatient visit was the 
most risky. Risk was especially high (75.3%) for patients with a 
LOS of less than 2 days and who had been hospitalized via 
outpatient visit. For the third stage, the department of treat- 
ment was the important factor, where the risk of readmission 
increased for the Department of Internal Medicine. In particu- 
lar, the risk of readmission was 80.6% if the patient was hospi- 
talized via outpatient visit in the Department of Internal Medi- 
cine with less than 2 days of stay. 

For patients with a LOS of more than 2 days, the important 
risk factor in the next stage was the category of principal diag- 
nosis (ICD-1 0), where the risk of readmission for patients with 
neoplasms with a LOS over 2 days was 51.2%. The important 
factor in the third stage was the LOS, with increased risk when 
the LOS was less than 5 days. If the category of principal diag- 
nosis was neoplasms with a LOS over 2 days but less than 5 
days, the risk of readmission was 69.2%. For categories other 
than neoplasms, the riskfactor of readmission was the number 
of outpatient visits within a year of the index hospitalization, 



where the risk increased if the number exceeded 10 times. 

DISCUSSION 

The most important variable in predicting the risk of read- 
mission was the LOS, where the risk was high when the LOS 
was less than two days (64.9%). Discharge within 24 or 36 hours 
of hospitalization was a risk increasing factor for various ages 
and disease categories due to decreased opportunity to dis- 
cover comorbidities or complications as well as difficulty in 
managing pain for patients discharged on the same day as 
ambulatory surgery [31-33 ]. 

For patients with a LOS of less than 2 days, the variable to 
predict the risk of readmission was the route of admission, 
where the risk of readmission was high (75.3%) if the LOS was 
less than 2 days and they were admitted via an outpatient vis- 
it. The severity and urgency for patients who come to the ER 
are high, but the consultation time is short. The medical staff 
of the ER leave a variety of possibilities open and makes diag- 
noses in greater detail than is done for outpatients, resulting 
in a reduced possibility of misdiagnosis or undiscovered ill- 
nesses [34]. In addition, intensive treatment is carried out in a 
relatively short time based on rapid diagnosis. In particular, 
based on this study, the diagnosis and respective treatment of 
comorbidity are likely to have been made faster for emergen- 
cy patients with a LOS less than 2 days than for those hospital- 
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ized via outpatient services. In other words, the risk of read- 
mission for patients who were discharged early (within 2 days 
of hospitalization), for the same reason mentioned earlier, is 
speculated to be higher than for those hospitalized via an out- 
patient visit than via the ER. 

The department of treatment was the most important vari- 
able in predicting the risk of readmission for patients with a 
LOS of less than 2 days and hospitalization via an outpatient 
visit. The risk of readmission was 80.6% for patients from the 
Department of Internal Medicine and 39.1% from the Depart- 
ment of Surgery. This study supports previous studies that 
concluded that the risk of readmission for patients from the 
Internal Medicine Department is generally higher due to the 
relatively high proportion of patients with chronic disease and 
complications [14,17,20]. 

Therefore, in order to prevent readmission, the appropriate- 
ness of discharge within 2 days of hospitalization should be 
considered with respect to undiscovered comorbidities and 
complications. It is especially important to provide appropri- 
ate examinations and rapid results for suspected disorders to 
patients admitted through an outpatient visit; moreover, close 
attention should be given to patients from the Department of 
Internal Medicine for potential comorbidities and complica- 
tions caused by chronic diseases. 

The category of principal diagnosis (ICD-10) was the most 
important variable for patients with a LOS of over 2 days. In 
previous studies, the title of the diagnosed disease was pro- 
posed as a factor of readmission based on the finding that the 
risk factor of readmission varies according to the specific dis- 
ease [35]. In a study of patients with pneumonia, congestive 
heart failure, and intracranial hemorrhage, analysis according 
to each type of disease correlated with the factors affecting 
readmission [14]. Hence, risk of readmission can be predicted 
more accurately by breaking down and analyzing the target of 
research according to more detailed types of diseases. The ti- 
tles of diseases are classified as "neoplasm" and "others" in the 
results of this study; that is because the risk factor is likely due 
to regular readmission for chemotherapy, a post-treatment, 
which is unavoidable [5]. 

LOS (standard: 5 days) was the key variable for predicting 
the risk of readmission for patients with LOS of more than 2 
days and disease in the "neoplasms" category, where the short- 
er the LOS was, the higher the risk of readmission (69.2%). Ac- 
cording to previous research, the risk of readmission within 1 
year of index hospitalization increased with an increase in LOS, 



in a study of patients over 14 years of age in training hospitals 
[20] while an increased risk of readmission within 28 days of 
discharge was noted for a decrease in LOS for patients in aca- 
demic hospitals [14]. In addition, LOS was not a significant fac- 
tor in a study of patients over 65 years who were readmitted 
within 60 days of discharge [36]. This difference appears to be 
due to the difference in the subjects of study and the defini- 
tion of the period from the day of discharge till readmission. 
For this study, the risk of readmission was higher when the 
LOS was relatively shorter with respect to the 5 day standard, 
which seems to be due to premature discharge without suffi- 
cient treatment. This phenomenon is caused by hospitals try- 
ing to reduce the LOS in order to increase bed turnover, which 
in turn increases profitability [37]. 

For patients classified in the "others" disease category, the 
factor of risk prediction was the number of outpatient visits 
within a year prior to the index hospitalization. Previous re- 
search shows that the risk of readmission is higher for patients 
who use medical services frequently; the risk of readmission 
increased with an increase in the frequency of hospitalization 
and use of ER within 6 months of index hospitalization as well 
as higher utilization of outpatient services within a year of in- 
dex hospitalization [14,1 9,28]. The present study supports the 
findings of the previous studies, as the risk of readmission 
evaluated by this study was 12.5% for patients who used out- 
patient services less than 1 0 times within one year of the index 
hospitalization, whereas it was 27.3% for those who used it 
more than 10 times. 

The first contribution of this study is that, unlike most prior 
studies that used a logistic regression, it used various methods 
to construct prediction models and then chose the model with 
the strongest predictive power. Second, it went beyond the 
simple task of identifying risk factors of readmission to finding 
the important predictive variables for each stage, thereby al- 
lowing a study of patterns and implications. 

Some limitations were, first, that the results of the study 
cannot be generalized, as the data used were of hospitalized 
patients from a single academic hospital. Second, the study 
could not include all the variables that reflect the state of the 
patients mentioned in prior research such as the severity of 
the disease, the title of accompanying diseases, and the pro- 
cess of providing healthcare services. Although the number of 
accompanying treatments and comorbidities were included 
as independent variables in order to make up for this limita- 
tion, it is difficult to directly identify the severity of the disease 
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and the process of providing medical services. Third, the deci- 
sion tree modeled the pattern of readmitted patients by cate- 
gorizing only 6 times, which means that it could not catego- 
rize according to all of the risk factors suggested in previous 
studies. In conclusion, the study provided results that help 
clarify a certain pattern in the way variables affect the identifi- 
cation of patients with risk of readmission, but it could not as- 
sess all the variables that affect readmission. It is recommend- 
ed that further studies include variables such as the severity of 
diseases and the process of provision of medical services. Also 
the risk factors should be studied with respect to detailed 
types of diseases. 
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